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Abstract. The problem of tag completion is to learn the missing tags 
of an image. In this paper, we propose to learn a tag scoring vector for 
each image by local linear learning. A local linear function is used in 
the neighborhood of each image to predict the tag scoring vectors of its 
neighboring images. We construct a unihed objective function for the 
learning of both tag scoring vectors and local linear function parame¬ 
ters. In the objective, we impose the learned tag scoring vectors to be 
consistent with the known associations to the tags of each image, and 
also minimize the prediction error of each local linear function, while 
reducing the complexity of each local function. The objective function 
is optimized by an alternate optimization strategy and gradient descent 
methods in an iterative algorithm. We compare the proposed algorithm 
against different state-of-the-art tag completion methods, and the results 
show its advantages. 
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1 Introduction 

Recently, social network has been a popular tool to share images. When a so¬ 
cial network user uploads an image, the image is usually associated with a tag/ 
keyword which is used to describe the semantic content of this image. The tags 
provided by the users are usually incomplete. Zhang et al. designed and im¬ 
plemented a fast motion detection mechanism for multimedia data on mobile 
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and embedded environment [30]. Recently, the problem image tag completion 
is proposed in the computer vision and machine learning communities to learn 
the missing tags of images [27,11,10,1,28,13,19]. This problem is defined as 
the problem of complete the missing elements of a tag vector of a given image 
automatically. 

In this paper, we investigate the problem of image tag completion, and pro¬ 
posed a novel and effective algorithm for this problem based on local linear 
learning. We propose a novel and effective tag completion method. Instead of 
completing the missing tag association elements of each image, we introduce a 
tag scoring vector to indicate the scores of assigning the image to the tags in a 
given tag set. We propose to study the tag scoring vector learning problem in the 
neighborhood of each image. For each image in the neighborhood, we propose to 
learn a linear function to predict a tag scoring vector from a visual feature vector 
of its corresponding image feature. We propose to minimize the perdition error 
measure by the squared (.% norm distance over each neighborhood, and also min¬ 
imize the squared ti norm of the linear function parameters. Besides the local 
linear learning, we also proposed to regularize the learning of tag scoring vectors 
by the available tags of each image. We construct a unified objective function 
to learn both the tag scoring vectors and the local linear functions. We develop 
an iterative algorithm to optimize the proposed problem. In each iteration of 
this algorithm, we update the tag scoring vectors and the local linear function 
parameters alternately. 

This rest parts of paper are organized as follows: in section 2, we introduced 
the proposed method. In section 3, we evaluate the proposed methods on some 
benchmark data sets. In section 4, the paper is concluded with future works. 


2 Proposed method 

We assume that we have a data set of n images, and their visual feature vec¬ 
tors are Xj|” =1 , where X; £ is the d-dimensional feature vector of the i- th 
image. We also assume that we have a set of m unique tags, and a tag vec¬ 
tor t i = [tii, - - • , t,: m ] T £ {+1,—l} m for the i-th image x*, where Uj = +1 
if the j-th tag is assigned to the i-th image, and —1, otherwise. In real-world 
applications, the tag vector of an image x* is usually incomplete, i.e., some ele¬ 
ments of t i are missing. We define a vector v, = [w*i, • • • , Uj m ] £ {1, 0} m , where 
Vij = 1 if tij is available, and 0 if tij is missing. We propose to learn a tag scoring 
vector tj = [tn, ■ ■ ■ ,tim} £ K m , where tij is the score of assigning the j-th tag 
to the i-th image. 

The set of the k nearest neighbor of each image x,; is denoted as A /j, and we 
assume that the tag scoring vector tj of a image Xj £ A/] can be predicted from 
its visual feature vector Xj using a local linear function jj(xj), 


tj < ft (xj ) = U,Xj. V j : Xj C Afi, 


(1) 
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where Wj £ K. mxd is the parameter of the local linear function. To learn the 
tag scoring vector and the local function parameters, we propose the following 
minimization problem, 


nun 


g (= E E ii*i - WiXj\\l+<*\w 

*=i \j:xjeNi 
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( 2 ) 


+ /3(t, - ti) T diag(vi)(ti - tj) 


where a and /3 are tradeoff parameters. The objective function < 5 i(tj|" =1 , Wj|" =1 ) 
in (2) is a summarization of three terms over all the images in the data set. The 
first term, J^j-.x-eM fl^i — WjXj Hi, is the prediction error term of the local linear 
predictor over the neighborhood of each image. The second, ||Wj|||, is to reduce 
the complexity of the local linear predictor. The last term, (tj — tj) T <iiag(vj)(tj — 
tj), is a regularization term to regularize the learning of tag scoring vectors by 
the incomplete tag vectors, so that the available tags are respected. To optimize 
the minimization problem in (2), we propose to use the alternate optimization 
strategy [4,12] in an iterative algorithm. 

— Optimization of t,|” =1 In each iteration, we optimize tj|” =1 one by one, 
and the minimization of (2) with respect to tj can be achieved with the 
following gradient descent update rule, 


new _ ^old 


gS7 ti g{t^ =1 ,Wi\U)\u=tfd, 


where Vt i (?(tj|" =1 , Wj|" =1 ) is the sub-gradient function of g(tj\j =1 , 
with respect to tj, 


(3) 

Wi\u), 


Vt i fl(t j |^ 1 ,Wj|r =1 ) = 2 ^2 (ti-W k -Xi) + 2Pdiag(vi)(ti-%), ^ 
and rj is the descent step. 

— Optimization of Wj|™ =1 In each iteration, we also optimized Wj|" =1 one 
by one. When W t is optimized, Wj\jjH are fixed. Gradient descent method 
is also employed to update Wj to minimize the objective in (2), 

Wr w = W° ld - rjVwMU, (5) 

where Vw i fl , (tj|" =1 , Wj|” =1 ) is the sub-gradient function with respect to Wj, 

VwwNti, Wj|? =1 ) = 2 (tj - W,x ( jxJ + 2f3W t . 

j-.-x.jeNi 


(6) 
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3 Experiments 

3.1 Setup 

In the experiments, we used two publicly accessed image-tag data sets, which are 
Corel5k data set [35,5,15] and IAPR TC12 data set [36,9,35]. In the Corel5k 
data set, there are 4,918 images, and 260 tages. We extract density feature, Harris 
shift feature, Harris Hue feature, RGB color feature, and HSV color feature as 
visual features for each image. Moreover, we remove 40% of the elements of 
the tag vectors to make the incomplete image tag vectors. In the IAPR TC12 
data set, there are 19,062 images, and 291 tags. We also remove 40% elements 
of the tag elements to construct the incomplete tag vectors. To evaluate the 
tag completion performances, we used the recall-precision curve as performance 
measure. We also use mean average precision (MAP) as a single performance 
measure. 

3.2 Results 

We compared the proposed method to several state-of-the-art tag completion 
methods, including tag matrix completion (TMC) [27], linear sparse reconstruc¬ 
tions (LSR) [10], tag completion by noisy matrix recovery (TCMR)[1], and tag 
completion via NMF (TC-NMF) [28]. The experimental result on two data sets 
are given in Fig. 1 and Fig. 2. From these figures, we can see that the proposed 
method LocTC performs best. Its recall-precision curve is closer to the top-right 
corner than any other methods, and its MAP is also higher than MAPs of other 
methods. 



(a) Recall-precision curve 


(b) MAP 


Fig. 1. Results of comparison to state-of-the-art methods on Corel5k data set 


In this section, we will study the sensitivity of the proposed algorithm to the 
two parameters, a and /3. The curves of a and /3 on different data sets are given 
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IAPRTC12 



IAPRTC12 



LocTC LSR TCMR TC-NMF TMC 


(a) Recall-precision curve 


(b) MAP 


Fig. 2. Results of comparison to state-of-the-art methods on IAPR TC12 data set 


in Fig. 3 and Fig. 4. From these figures, we can see that the performances are 
stable to different valuse of both a and /?. 


Corel5k 
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0.01 0.1 1 100 1000 10000 
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(b) P 


Fig. 3. Parameter sensitivity curve on Corel5k data set. 


4 Conclusion and future works 

In this paper, we study the problem of tag completion, and proposed a novel 
algorithm for this problem. We proposed to learn the tags of images in the 
neighborhood of each image. A local linear function is designed to predict the 
tag scoring vectors of images in each neighborhood, and the prediction function 
parameter is learned jointly with the tag scoring vectors. The proposed method 
















6 


J. Wang, et al. 


IAPRTC12 



0.01 0.1 1 100 1000 10000 
alpha 


IAPRTC12 


0.2 

CL 

< 

0.1 


0 1 —‘-‘-‘-‘-‘-‘— 

0.01 0.1 1 100 1000 10000 

beta 


(a) a 


(b)£ 


Fig. 4. Parameter sensitivity curve on IAPR TC12 data set. 


is compared to state-of-the-art tag completion algorithms, and the results show 
that the proposed algorithm outperforms the compared methods. In the future, 
we will study how to incorporate these connections into our model and learn 
more effective tags. In this paper, we used one single local function for each 
neighborhood, and in the future, we will use more than than regularization to 
regularized the learning of tags [20,21], such as usage of wavelet functions to 
construct the local function [14]. Moreover, correntropy can also be considered 
as a alternative loss function to construct the local learning problem [22, 29,6, 
39,16]. In the future, we also plan to extend the proposed algorithm for comple¬ 
tion of tags of large scale image data set by using high performance computing 
technology [41,26,33,34,37,38,2,30,8,40,7,23,25,24], and completion of tags 
of gene/protein functions of bioinformatics problems [3,31,32,3,17,18]. 
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